{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f6539807-d2be-47b6-a39e-aaacb1e6e0bb",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2403e87-604e-44c6-8d9e-6c5806bc4cd7",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align: center;\">Evaluating Rogue Score</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef54d59-295e-4f15-a35f-33f4e86ecdd2",
   "metadata": {},
   "source": [
    "**What is Rogue Score?**: The Rouge score is a metric used to measure the similarity between two pieces of text. It's Commonly used in tasks like text summarization, machine translation, and text generation to assess how well the generated text aligns with a predefined reference.\n",
    "\n",
    "For example, for the reference text 'The cat sat on the mat,' the LLM might generate a similar one, such as 'A cat is sitting on a mat,' or it could generate something completely different like 'The dog is lying on the carpet.' The Rouge Score helps us measure how closely these sentences align. A high Rouge Score indicates a close match in meaning, signifying the effectiveness of the language model in faithfully reproducing the original sentence. This is useful in real life for tasks such as verifying if an AI-generated news headline genuinely expresses the same idea as the original news article.\"\n",
    "\n",
    "**Data schema**: The data schema required for this evaluation is as follows:\n",
    "\n",
    "| Column Name | Description |\n",
    "| ----------- | ----------- |\n",
    "| text-generated | The column containing the generated text for comparison. |\n",
    "| text_source | The column containing the source or reference text. |\n",
    "| rouge_score | The column that will store the Rouge scores after the evaluation. |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76008cf6-dbd3-44bc-8556-86278868e746",
   "metadata": {},
   "source": [
    "Install UpTrain by running 'pip install uptrain'\n",
    "Install polars by running 'pip install polars'\n",
    "Install rouge-score by running 'pip install rouge-score'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "90f4ff7d-10b9-482e-aa92-85c64ffc4057",
   "metadata": {},
   "outputs": [],
   "source": [
    "import polars as pl\n",
    "from uptrain.operators import RougeScore\n",
    "\n",
    "OPENAI_API_KEY = \"sk-*****************\"  # Insert your OpenAI key here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first evaluate some data with high rouge data (with respect to the source)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "11a472a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "high_rouge_df = pl.DataFrame({\n",
    "    \"text_generated\": [\n",
    "        \"The company is focused on technological innovation.\",\n",
    "        \"Climate change poses significant challenges for our planet.\",\n",
    "        \"Renewable energy sources play a key role in reducing environmental impact.\",\n",
    "        \"Global collaboration is essential for addressing complex issues.\",\n",
    "        \"Advancements in healthcare technology enhance patient care.\"\n",
    "    ],\n",
    "    \"text_source\": [\n",
    "        \"The company is dedicated to technological innovation.\",\n",
    "        \"Climate change presents considerable challenges for our planet.\",\n",
    "        \"Renewable energy is crucial for reducing environmental impact.\",\n",
    "        \"Global cooperation is necessary for tackling complex issues.\",\n",
    "        \"Progress in medical technology improves patient care.\"\n",
    "    ]\n",
    "})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d15fb087-3ed2-442f-9b23-35c7dbc6128e",
   "metadata": {},
   "outputs": [],
   "source": [
    "rouge_op = RougeScore(score_type=\"f1\")\n",
    "rouge_score = rouge_op.run(high_rouge_df)[\"output\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0a950849-0348-450f-add0-cea2c1423525",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 3)\n",
      "┌───────────────────────────────────┬───────────────────────────────────┬─────────────┐\n",
      "│ text_generated                    ┆ text_source                       ┆ rouge_score │\n",
      "│ ---                               ┆ ---                               ┆ ---         │\n",
      "│ str                               ┆ str                               ┆ i64         │\n",
      "╞═══════════════════════════════════╪═══════════════════════════════════╪═════════════╡\n",
      "│ The company is focused on techno… ┆ The company is dedicated to tech… ┆ 71          │\n",
      "│ Climate change poses significant… ┆ Climate change presents consider… ┆ 75          │\n",
      "│ Renewable energy sources play a … ┆ Renewable energy is crucial for … ┆ 52          │\n",
      "│ Global collaboration is essentia… ┆ Global cooperation is necessary … ┆ 62          │\n",
      "│ Advancements in healthcare techn… ┆ Progress in medical technology i… ┆ 57          │\n",
      "└───────────────────────────────────┴───────────────────────────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "print(rouge_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's evaluate some data with low rouge data (with respect to the source)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "low_rouge_df = pl.DataFrame({\n",
    "    \"text_generated\": [\n",
    "        \"The cat sat on the mat.\",\n",
    "        \"I like to eat pizza for dinner.\",\n",
    "        \"Summer is my favorite season.\",\n",
    "        \"Programming is an interesting skill to learn.\",\n",
    "        \"Mountains are a breathtaking natural wonder.\"\n",
    "    ],\n",
    "    \"text_source\": [\n",
    "        \"A cat is sitting on a mat.\",\n",
    "        \"Pizza is my preferred choice for dinner.\",\n",
    "        \"Winter is the season I enjoy the most.\",\n",
    "        \"Learning to program can be fascinating.\",\n",
    "        \"The beauty of nature is evident in mountains.\"\n",
    "    ]\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "rouge_op = RougeScore(score_type=\"f1\")\n",
    "rouge_score = rouge_op.run(low_rouge_df)[\"output\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (5, 3)\n",
      "┌───────────────────────────────────┬───────────────────────────────────┬─────────────┐\n",
      "│ text_generated                    ┆ text_source                       ┆ rouge_score │\n",
      "│ ---                               ┆ ---                               ┆ ---         │\n",
      "│ str                               ┆ str                               ┆ i64         │\n",
      "╞═══════════════════════════════════╪═══════════════════════════════════╪═════════════╡\n",
      "│ The cat sat on the mat.           ┆ A cat is sitting on a mat.        ┆ 46          │\n",
      "│ I like to eat pizza for dinner.   ┆ Pizza is my preferred choice for… ┆ 42          │\n",
      "│ Summer is my favorite season.     ┆ Winter is the season I enjoy the… ┆ 30          │\n",
      "│ Programming is an interesting sk… ┆ Learning to program can be fasci… ┆ 15          │\n",
      "│ Mountains are a breathtaking nat… ┆ The beauty of nature is evident … ┆ 14          │\n",
      "└───────────────────────────────────┴───────────────────────────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "print(rouge_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "346c0e30",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
